

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  <meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />

  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>OSD Service &mdash; Ceph Documentation</title>
  

  
  <link rel="stylesheet" href="../../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/graphviz.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/css/custom.css" type="text/css" />

  
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../../" src="../../../_static/documentation_options.js"></script>
        <script src="../../../_static/jquery.js"></script>
        <script src="../../../_static/_sphinx_javascript_frameworks_compat.js"></script>
        <script data-url_root="../../../" id="documentation_options" src="../../../_static/documentation_options.js"></script>
        <script src="../../../_static/doctools.js"></script>
        <script src="../../../_static/sphinx_highlight.js"></script>
    
    <script type="text/javascript" src="../../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../../genindex/" />
    <link rel="search" title="Search" href="../../../search/" />
    <link rel="next" title="RGW Service" href="../rgw/" />
    <link rel="prev" title="MGR Service" href="../mgr/" /> 
</head>

<body class="wy-body-for-nav">

   
  <header class="top-bar">
    <div role="navigation" aria-label="Page navigation">
  <ul class="wy-breadcrumbs">
      <li><a href="../../../" class="icon icon-home" aria-label="Home"></a></li>
          <li class="breadcrumb-item"><a href="../../">Cephadm</a></li>
          <li class="breadcrumb-item"><a href="../">Service Management</a></li>
      <li class="breadcrumb-item active">OSD Service</li>
      <li class="wy-breadcrumbs-aside">
            <a href="../../../_sources/cephadm/services/osd.rst.txt" rel="nofollow"> View page source</a>
      </li>
  </ul>
  <hr/>
</div>
  </header>
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #eee" >
          

          
            <a href="../../../" class="icon icon-home"> Ceph
          

          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../../search/" method="get">
    <input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../../start/">Ceph 简介</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../install/">安装 Ceph</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../../">Cephadm</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../../compatibility/">Compatibility and Stability</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../install/">部署个全新的 Ceph 集群</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../adoption/">现有集群切换到 cephadm</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../host-management/">Host Management</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../">Service Management</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="../mon/">MON Service</a></li>
<li class="toctree-l3"><a class="reference internal" href="../mgr/">MGR Service</a></li>
<li class="toctree-l3 current"><a class="current reference internal" href="#">OSD Service</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#list-devices">List Devices</a></li>
<li class="toctree-l4"><a class="reference internal" href="#retrieve-exact-size-of-block-devices">Retrieve Exact Size of Block Devices</a></li>
<li class="toctree-l4"><a class="reference internal" href="#deploy-osds">Deploy OSDs</a></li>
<li class="toctree-l4"><a class="reference internal" href="#remove-an-osd">Remove an OSD</a></li>
<li class="toctree-l4"><a class="reference internal" href="#automatically-tuning-osd-memory">Automatically tuning OSD memory</a></li>
<li class="toctree-l4"><a class="reference internal" href="#advanced-osd-service-specifications">Advanced OSD Service Specifications</a></li>
<li class="toctree-l4"><a class="reference internal" href="#examples">Examples</a></li>
<li class="toctree-l4"><a class="reference internal" href="#activate-existing-osds">Activate existing OSDs</a></li>
<li class="toctree-l4"><a class="reference internal" href="#further-reading">Further Reading</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../rgw/">RGW Service</a></li>
<li class="toctree-l3"><a class="reference internal" href="../mds/">MDS Service</a></li>
<li class="toctree-l3"><a class="reference internal" href="../nfs/">NFS Service</a></li>
<li class="toctree-l3"><a class="reference internal" href="../iscsi/">iSCSI Service</a></li>
<li class="toctree-l3"><a class="reference internal" href="../custom-container/">Custom Container Service</a></li>
<li class="toctree-l3"><a class="reference internal" href="../monitoring/">Monitoring Services</a></li>
<li class="toctree-l3"><a class="reference internal" href="../snmp-gateway/">SNMP Gateway Service</a></li>
<li class="toctree-l3"><a class="reference internal" href="../tracing/">如何追踪各服务</a></li>
<li class="toctree-l3"><a class="reference internal" href="../smb/">SMB Service</a></li>
<li class="toctree-l3"><a class="reference internal" href="../mgmt-gateway/">Management Gateway</a></li>
<li class="toctree-l3"><a class="reference internal" href="../oauth2-proxy/">OAuth2 Proxy</a></li>
<li class="toctree-l3"><a class="reference internal" href="../#service-status">Service Status</a></li>
<li class="toctree-l3"><a class="reference internal" href="../#daemon-status">Daemon Status</a></li>
<li class="toctree-l3"><a class="reference internal" href="../#service-specification">Service Specification</a></li>
<li class="toctree-l3"><a class="reference internal" href="../#daemon-placement">Daemon Placement</a></li>
<li class="toctree-l3"><a class="reference internal" href="../#extra-container-arguments">Extra Container Arguments</a></li>
<li class="toctree-l3"><a class="reference internal" href="../#extra-entrypoint-arguments">Extra Entrypoint Arguments</a></li>
<li class="toctree-l3"><a class="reference internal" href="../#custom-config-files">Custom Config Files</a></li>
<li class="toctree-l3"><a class="reference internal" href="../#removing-a-service">Removing a Service</a></li>
<li class="toctree-l3"><a class="reference internal" href="../#disabling-automatic-deployment-of-daemons">Disabling automatic deployment of daemons</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../certmgr/">Certificate Management</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../upgrade/">升级 Ceph</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../operations/">Cephadm operations</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../client-setup/">Client Setup</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../troubleshooting/">Troubleshooting</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../../dev/cephadm/">Cephadm Feature Planning</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../../rados/">Ceph 存储集群</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../cephfs/">Ceph 文件系统</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../rbd/">Ceph 块设备</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../radosgw/">Ceph 对象网关</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../mgr/">Ceph 管理器守护进程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../mgr/dashboard/">Ceph 仪表盘</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../monitoring/">监控概览</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../api/">API 文档</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../architecture/">体系结构</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../dev/developer_guide/">开发者指南</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../dev/internals/">Ceph 内幕</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../governance/">项目管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../foundation/">Ceph 基金会</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../ceph-volume/">ceph-volume</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../releases/general/">Ceph 版本（总目录）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../releases/">Ceph 版本（索引）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../security/">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../hardware-monitoring/">硬件监控</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../glossary/">Ceph 术语</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../jaegertracing/">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../translation_cn/">中文版翻译资源</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../../">Ceph</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
<div id="dev-warning" class="admonition note">
  <p class="first admonition-title">Notice</p>
  <p class="last">This document is for a development version of Ceph.</p>
</div>
  <div id="docubetter" align="right" style="padding: 5px; font-weight: bold;">
    <a href="https://pad.ceph.com/p/Report_Documentation_Bugs">Report a Documentation Bug</a>
  </div>

  
  <section id="osd-service">
<h1>OSD Service<a class="headerlink" href="#osd-service" title="Permalink to this heading"></a></h1>
<section id="list-devices">
<h2>List Devices<a class="headerlink" href="#list-devices" title="Permalink to this heading"></a></h2>
<p><code class="docutils literal notranslate"><span class="pre">ceph-volume</span></code> scans each host in the cluster periodically in order
to determine which devices are present and whether they are eligible to be
used as OSDs.</p>
<p>To print a list of devices discovered by <code class="docutils literal notranslate"><span class="pre">cephadm</span></code>, run this command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><style type="text/css">
span.prompt1:before {
  content: "# ";
}
</style><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>device<span class="w"> </span>ls<span class="w"> </span><span class="o">[</span>--hostname<span class="o">=</span>...<span class="o">]</span><span class="w"> </span><span class="o">[</span>--wide<span class="o">]</span><span class="w"> </span><span class="o">[</span>--refresh<span class="o">]</span></span>
</pre></div></div><p>Example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">Hostname</span>  <span class="n">Path</span>      <span class="n">Type</span>  <span class="n">Serial</span>              <span class="n">Size</span>   <span class="n">Health</span>   <span class="n">Ident</span>  <span class="n">Fault</span>  <span class="n">Available</span>
<span class="n">srv</span><span class="o">-</span><span class="mi">01</span>    <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdb</span>  <span class="n">hdd</span>   <span class="mi">15</span><span class="n">P0A0YFFRD6</span>         <span class="mi">300</span><span class="n">G</span>  <span class="n">Unknown</span>  <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">No</span>
<span class="n">srv</span><span class="o">-</span><span class="mi">01</span>    <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdc</span>  <span class="n">hdd</span>   <span class="mi">15</span><span class="n">R0A08WFRD6</span>         <span class="mi">300</span><span class="n">G</span>  <span class="n">Unknown</span>  <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">No</span>
<span class="n">srv</span><span class="o">-</span><span class="mi">01</span>    <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdd</span>  <span class="n">hdd</span>   <span class="mi">15</span><span class="n">R0A07DFRD6</span>         <span class="mi">300</span><span class="n">G</span>  <span class="n">Unknown</span>  <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">No</span>
<span class="n">srv</span><span class="o">-</span><span class="mi">01</span>    <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sde</span>  <span class="n">hdd</span>   <span class="mi">15</span><span class="n">P0A0QDFRD6</span>         <span class="mi">300</span><span class="n">G</span>  <span class="n">Unknown</span>  <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">No</span>
<span class="n">srv</span><span class="o">-</span><span class="mi">02</span>    <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdb</span>  <span class="n">hdd</span>   <span class="mi">15</span><span class="n">R0A033FRD6</span>         <span class="mi">300</span><span class="n">G</span>  <span class="n">Unknown</span>  <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">No</span>
<span class="n">srv</span><span class="o">-</span><span class="mi">02</span>    <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdc</span>  <span class="n">hdd</span>   <span class="mi">15</span><span class="n">R0A05XFRD6</span>         <span class="mi">300</span><span class="n">G</span>  <span class="n">Unknown</span>  <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">No</span>
<span class="n">srv</span><span class="o">-</span><span class="mi">02</span>    <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sde</span>  <span class="n">hdd</span>   <span class="mi">15</span><span class="n">R0A0ANFRD6</span>         <span class="mi">300</span><span class="n">G</span>  <span class="n">Unknown</span>  <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">No</span>
<span class="n">srv</span><span class="o">-</span><span class="mi">02</span>    <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdf</span>  <span class="n">hdd</span>   <span class="mi">15</span><span class="n">R0A06EFRD6</span>         <span class="mi">300</span><span class="n">G</span>  <span class="n">Unknown</span>  <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">No</span>
<span class="n">srv</span><span class="o">-</span><span class="mi">03</span>    <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdb</span>  <span class="n">hdd</span>   <span class="mi">15</span><span class="n">R0A0OGFRD6</span>         <span class="mi">300</span><span class="n">G</span>  <span class="n">Unknown</span>  <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">No</span>
<span class="n">srv</span><span class="o">-</span><span class="mi">03</span>    <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdc</span>  <span class="n">hdd</span>   <span class="mi">15</span><span class="n">R0A0P7FRD6</span>         <span class="mi">300</span><span class="n">G</span>  <span class="n">Unknown</span>  <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">No</span>
<span class="n">srv</span><span class="o">-</span><span class="mi">03</span>    <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdd</span>  <span class="n">hdd</span>   <span class="mi">15</span><span class="n">R0A0O7FRD6</span>         <span class="mi">300</span><span class="n">G</span>  <span class="n">Unknown</span>  <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">N</span><span class="o">/</span><span class="n">A</span>    <span class="n">No</span>
</pre></div>
</div>
<p>The <code class="docutils literal notranslate"><span class="pre">--wide</span></code> option shows device details,
including any reasons that the device might not be eligible for use as an OSD.</p>
<p>In the above example you can see fields named <code class="docutils literal notranslate"><span class="pre">Health</span></code>, <code class="docutils literal notranslate"><span class="pre">Ident</span></code>, and <code class="docutils literal notranslate"><span class="pre">Fault</span></code>.
This information is provided by integration with <a class="reference external" href="https://github.com/libstorage/libstoragemgmt">libstoragemgmt</a>. By default,
this integration is disabled because <a class="reference external" href="https://github.com/libstorage/libstoragemgmt">libstoragemgmt</a> may not be 100%
compatible with your hardware.  To direct Ceph to include these fields,
enable <code class="docutils literal notranslate"><span class="pre">cephadm</span></code>’s “enhanced device scan” option as follows:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>config<span class="w"> </span><span class="nb">set</span><span class="w"> </span>mgr<span class="w"> </span>mgr/cephadm/device_enhanced_scan<span class="w"> </span><span class="nb">true</span></span>
</pre></div></div><div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>Although the <code class="docutils literal notranslate"><span class="pre">libstoragemgmt</span></code> library performs standard SCSI inquiry calls,
there is no guarantee that your hardware and firmware properly implement these standards.
This can lead to erratic behaviour and even bus resets on some older
hardware. It is therefore recommended that, before enabling this feature,
you first test your hardware’s compatibility with <code class="docutils literal notranslate"><span class="pre">libstoragemgmt</span></code> to avoid
unplanned interruptions to services.</p>
<p>There are a number of ways to test compatibility, but the simplest is
to use the cephadm shell to call <code class="docutils literal notranslate"><span class="pre">libstoragemgmt</span></code> directly: <code class="docutils literal notranslate"><span class="pre">cephadm</span> <span class="pre">shell</span>
<span class="pre">lsmcli</span> <span class="pre">ldl</span></code>. If your hardware is supported you should see something like
this:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">Path</span>     <span class="o">|</span> <span class="n">SCSI</span> <span class="n">VPD</span> <span class="mh">0x83</span>    <span class="o">|</span> <span class="n">Link</span> <span class="n">Type</span> <span class="o">|</span> <span class="n">Serial</span> <span class="n">Number</span>      <span class="o">|</span> <span class="n">Health</span> <span class="n">Status</span>
<span class="o">----------------------------------------------------------------------------</span>
<span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sda</span> <span class="o">|</span> <span class="mi">50000396082</span><span class="n">ba631</span> <span class="o">|</span> <span class="n">SAS</span>       <span class="o">|</span> <span class="mi">15</span><span class="n">P0A0R0FRD6</span>       <span class="o">|</span> <span class="n">Good</span>
<span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdb</span> <span class="o">|</span> <span class="mi">50000396082</span><span class="n">bbbf9</span> <span class="o">|</span> <span class="n">SAS</span>       <span class="o">|</span> <span class="mi">15</span><span class="n">P0A0YFFRD6</span>       <span class="o">|</span> <span class="n">Good</span>
</pre></div>
</div>
</div>
<p>After enabling <code class="docutils literal notranslate"><span class="pre">libstoragemgmt</span></code> support, the output will look something
like this:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># ceph orch device ls</span>
<span class="n">Hostname</span>   <span class="n">Path</span>      <span class="n">Type</span>  <span class="n">Serial</span>              <span class="n">Size</span>   <span class="n">Health</span>   <span class="n">Ident</span>  <span class="n">Fault</span>  <span class="n">Available</span>
<span class="n">srv</span><span class="o">-</span><span class="mi">01</span>     <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdb</span>  <span class="n">hdd</span>   <span class="mi">15</span><span class="n">P0A0YFFRD6</span>         <span class="mi">300</span><span class="n">G</span>  <span class="n">Good</span>     <span class="n">Off</span>    <span class="n">Off</span>    <span class="n">No</span>
<span class="n">srv</span><span class="o">-</span><span class="mi">01</span>     <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdc</span>  <span class="n">hdd</span>   <span class="mi">15</span><span class="n">R0A08WFRD6</span>         <span class="mi">300</span><span class="n">G</span>  <span class="n">Good</span>     <span class="n">Off</span>    <span class="n">Off</span>    <span class="n">No</span>
<span class="p">:</span>
</pre></div>
</div>
<p>In this example, <code class="docutils literal notranslate"><span class="pre">libstoragemgmt</span></code> has confirmed the health of the drives and the ability to
interact with the identification and fault LEDs on the drive enclosures. For further
information about interacting with these LEDs, refer to <a class="reference internal" href="../../../rados/operations/devices/#devices"><span class="std std-ref">设备管理</span></a>.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>The current release of <cite>libstoragemgmt`</cite> (1.8.8) supports SCSI, SAS, and SATA based
local drives only. There is no official support for NVMe devices (PCIe), SAN LUNs,
or exotic/complex metadevices.</p>
</div>
</section>
<section id="retrieve-exact-size-of-block-devices">
<h2>Retrieve Exact Size of Block Devices<a class="headerlink" href="#retrieve-exact-size-of-block-devices" title="Permalink to this heading"></a></h2>
<p>Run a command of the following form to discover the exact size of a block
device. The value returned here is used by the orchestrator when filtering based
on size:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">cephadm<span class="w"> </span>shell<span class="w"> </span>ceph-volume<span class="w"> </span>inventory<span class="w"> </span>&lt;/dev/sda&gt;<span class="w"> </span>--format<span class="w"> </span>json<span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span>.sys_api.human_readable_size</span>
</pre></div></div><p>The exact size in GB is the size reported in TB, multiplied by 1024.</p>
<section id="example">
<h3>Example<a class="headerlink" href="#example" title="Permalink to this heading"></a></h3>
<p>The following provides a specific example of this command based upon the
general form of the command above:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">cephadm<span class="w"> </span>shell<span class="w"> </span>ceph-volume<span class="w"> </span>inventory<span class="w"> </span>/dev/sdc<span class="w"> </span>--format<span class="w"> </span>json<span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span>.sys_api.human_readable_size</span>
</pre></div></div><div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="s2">&quot;3.64 TB&quot;</span>
</pre></div>
</div>
<p>This indicates that the exact device size is 3.64 TB, or 3727.36 GB.</p>
<p>This procedure was developed by Frédéric Nass. See <a class="reference external" href="https://lists.ceph.io/hyperkitty/list/ceph-users&#64;ceph.io/message/5BAAYFCQAZZDRSNCUPCVBNEPGJDARRZA/">this thread on the
[ceph-users] mailing list</a>
for discussion of this matter.</p>
</section>
</section>
<section id="deploy-osds">
<span id="cephadm-deploy-osds"></span><h2>Deploy OSDs<a class="headerlink" href="#deploy-osds" title="Permalink to this heading"></a></h2>
<section id="listing-storage-devices">
<h3>Listing Storage Devices<a class="headerlink" href="#listing-storage-devices" title="Permalink to this heading"></a></h3>
<p>In order to deploy an OSD, there must be an available storage device or devices on
which the OSD will be deployed.</p>
<p>Run this command to display an inventory of storage devices on all cluster hosts:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>device<span class="w"> </span>ls</span>
</pre></div></div><p>A storage device is considered _available_ if all of the following
conditions are met:</p>
<ul class="simple">
<li><p>The device must have no partitions.</p></li>
<li><p>The device must not have any LVM state.</p></li>
<li><p>The device must not be mounted.</p></li>
<li><p>The device must not contain a file system.</p></li>
<li><p>The device must not contain a Ceph BlueStore OSD.</p></li>
<li><p>The device must be larger than 5 GB.</p></li>
</ul>
<p>Ceph will not provision an OSD on a device that is not _available_.</p>
</section>
<section id="creating-new-osds">
<h3>Creating New OSDs<a class="headerlink" href="#creating-new-osds" title="Permalink to this heading"></a></h3>
<p>There are multiple ways to create new OSDs:</p>
<ul>
<li><p>Consume any available and unused storage device:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>apply<span class="w"> </span>osd<span class="w"> </span>--all-available-devices</span>
</pre></div></div></li>
<li><p>Create an OSD from a specific device on a specific host:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>daemon<span class="w"> </span>add<span class="w"> </span>osd<span class="w"> </span>*&lt;host&gt;*:*&lt;device-path&gt;*</span>
</pre></div></div><p>For example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>daemon<span class="w"> </span>add<span class="w"> </span>osd<span class="w"> </span>host1:/dev/sdb</span>
</pre></div></div></li>
<li><p>Advanced OSD creation from specific devices on a specific host:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>daemon<span class="w"> </span>add<span class="w"> </span>osd<span class="w"> </span>host1:data_devices<span class="o">=</span>/dev/sda,/dev/sdb,db_devices<span class="o">=</span>/dev/sdc,osds_per_device<span class="o">=</span><span class="m">2</span></span>
</pre></div></div></li>
<li><p>Create an OSD on a specific LVM logical volume on a specific host:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>daemon<span class="w"> </span>add<span class="w"> </span>osd<span class="w"> </span>*&lt;host&gt;*:*&lt;lvm-path&gt;*</span>
</pre></div></div><p>For example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>daemon<span class="w"> </span>add<span class="w"> </span>osd<span class="w"> </span>host1:/dev/vg_osd/lvm_osd1701</span>
</pre></div></div></li>
<li><p>You can use <a class="reference internal" href="#drivegroups"><span class="std std-ref">Advanced OSD Service Specifications</span></a> to categorize devices based on their
properties. This is useful to clarify which
devices are available to consume. Properties include device type (SSD or
HDD), device model names, size, and the hosts on which the devices exist:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>apply<span class="w"> </span>-i<span class="w"> </span>spec.yml</span>
</pre></div></div></li>
</ul>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>When deploying new OSDs with <code class="docutils literal notranslate"><span class="pre">cephadm</span></code>, ensure that the <code class="docutils literal notranslate"><span class="pre">ceph-osd</span></code> package is not installed on the target host. If it is installed, conflicts may arise in the management and control of the OSD that may lead to errors or unexpected behavior.</p>
</div>
<ul>
<li><p>OSDs created via <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span> <span class="pre">daemon</span> <span class="pre">add</span></code> are by default not added to the orchestrator’s OSD service. To attach an OSD to a different, existing OSD service, issue a command of the following form:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><style type="text/css">
span.prompt2:before {
  content: "* ";
}
</style><span class="prompt2">ceph<span class="w"> </span>orch<span class="w"> </span>osd<span class="w"> </span>set-spec-affinity<span class="w"> </span>&lt;service_name&gt;<span class="w"> </span>&lt;osd_id<span class="o">(</span>s<span class="o">)</span>&gt;</span>
</pre></div></div><p>For example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>osd<span class="w"> </span>set-spec-affinity<span class="w"> </span>osd.default_drive_group<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">1</span></span>
</pre></div></div></li>
</ul>
</section>
<section id="dry-run">
<h3>Dry Run<a class="headerlink" href="#dry-run" title="Permalink to this heading"></a></h3>
<p>The <code class="docutils literal notranslate"><span class="pre">--dry-run</span></code> flag causes the orchestrator to present a preview of what
will happen without actually creating the OSDs.</p>
<p>For example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>apply<span class="w"> </span>osd<span class="w"> </span>--all-available-devices<span class="w"> </span>--dry-run</span>
</pre></div></div><div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">NAME</span>                  <span class="n">HOST</span>  <span class="n">DATA</span>      <span class="n">DB</span>  <span class="n">WAL</span>
<span class="nb">all</span><span class="o">-</span><span class="n">available</span><span class="o">-</span><span class="n">devices</span> <span class="n">node1</span> <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">vdb</span>  <span class="o">-</span>   <span class="o">-</span>
<span class="nb">all</span><span class="o">-</span><span class="n">available</span><span class="o">-</span><span class="n">devices</span> <span class="n">node2</span> <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">vdc</span>  <span class="o">-</span>   <span class="o">-</span>
<span class="nb">all</span><span class="o">-</span><span class="n">available</span><span class="o">-</span><span class="n">devices</span> <span class="n">node3</span> <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">vdd</span>  <span class="o">-</span>   <span class="o">-</span>
</pre></div>
</div>
</section>
<section id="declarative-state">
<span id="cephadm-osd-declarative"></span><h3>Declarative State<a class="headerlink" href="#declarative-state" title="Permalink to this heading"></a></h3>
<p>The effect of <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span> <span class="pre">apply</span></code> is persistent. This means that drives that
are added to the system after the <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span> <span class="pre">apply</span></code> command completes will be
automatically detected and added to the cluster as specified.  It also means that drives that
become available (e.g. by zapping) after the <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span> <span class="pre">apply</span></code>
command completes will be automatically found and added to the cluster.</p>
<p>We will examine the effects of the following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>apply<span class="w"> </span>osd<span class="w"> </span>--all-available-devices</span>
</pre></div></div><p>After running the above command:</p>
<ul class="simple">
<li><p>When you add new drives to the cluster, they will automatically be used to
create new OSDs.</p></li>
<li><p>When you remove an OSD and clean the LVM physical volume, a new OSD will be
created automatically.</p></li>
</ul>
<p>If you want to avoid this behavior (disable automatic creation of OSD on available devices), use the <code class="docutils literal notranslate"><span class="pre">unmanaged</span></code> parameter:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>apply<span class="w"> </span>osd<span class="w"> </span>--all-available-devices<span class="w"> </span>--unmanaged<span class="o">=</span><span class="nb">true</span></span>
</pre></div></div><div class="admonition note">
<p class="admonition-title">Note</p>
<p>Keep these three facts in mind:</p>
<ul class="simple">
<li><p>The default behavior of <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span> <span class="pre">apply</span></code> causes <code class="docutils literal notranslate"><span class="pre">cephadm</span></code> to constantly reconcile. This means that <code class="docutils literal notranslate"><span class="pre">cephadm</span></code> creates OSDs as soon as new drives are detected.</p></li>
<li><p>Setting <code class="docutils literal notranslate"><span class="pre">unmanaged:</span> <span class="pre">True</span></code> disables the creation of OSDs. If <code class="docutils literal notranslate"><span class="pre">unmanaged:</span> <span class="pre">True</span></code> is set, nothing will happen even if you apply a new OSD service.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span> <span class="pre">daemon</span> <span class="pre">add</span></code> creates OSDs, but does not add an OSD service.</p></li>
</ul>
</div>
<ul class="simple">
<li><p>For more on <code class="docutils literal notranslate"><span class="pre">cephadm</span></code>, see also <a class="reference internal" href="../#cephadm-spec-unmanaged"><span class="std std-ref">Disabling automatic deployment of daemons</span></a>.</p></li>
</ul>
</section>
</section>
<section id="remove-an-osd">
<span id="cephadm-osd-removal"></span><h2>Remove an OSD<a class="headerlink" href="#remove-an-osd" title="Permalink to this heading"></a></h2>
<p>Removing an OSD from a cluster involves two steps:</p>
<ol class="arabic simple">
<li><p>evacuating all placement groups (PGs) from the OSD</p></li>
<li><p>removing the PG-free OSD from the cluster</p></li>
</ol>
<p>The following command performs these two steps:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>osd<span class="w"> </span>rm<span class="w"> </span>&lt;osd_id<span class="o">(</span>s<span class="o">)</span>&gt;<span class="w"> </span><span class="o">[</span>--replace<span class="o">]</span><span class="w"> </span><span class="o">[</span>--force<span class="o">]</span></span>
</pre></div></div><p>Example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>osd<span class="w"> </span>rm<span class="w"> </span><span class="m">0</span></span>
</pre></div></div><p>Expected output:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">Scheduled</span> <span class="n">OSD</span><span class="p">(</span><span class="n">s</span><span class="p">)</span> <span class="k">for</span> <span class="n">removal</span>
</pre></div>
</div>
<p>OSDs that are not safe to destroy will be rejected.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>After removing OSDs, if the OSDs’ drives
become available, <code class="docutils literal notranslate"><span class="pre">cephadm</span></code> may automatically try to deploy more OSDs
on these drives if they match an existing drivegroup spec. If you deployed
the OSDs you are removing with a spec and don’t want any new OSDs deployed on
the drives after removal, it’s best to modify the drivegroup spec before removal.
Either set <code class="docutils literal notranslate"><span class="pre">unmanaged:</span> <span class="pre">true</span></code> to stop it from picking up new drives,
or modify it in some way that it no longer matches the drives used for the
OSDs you wish to remove. Then re-apply the spec. For more info on drivegroup
specs see <a class="reference internal" href="#drivegroups"><span class="std std-ref">Advanced OSD Service Specifications</span></a>. For more info on the declarative nature of
<code class="docutils literal notranslate"><span class="pre">cephadm</span></code> in reference to deploying OSDs, see <a class="reference internal" href="#cephadm-osd-declarative"><span class="std std-ref">Declarative State</span></a></p>
</div>
<section id="monitoring-osd-state-during-osd-removal">
<h3>Monitoring OSD State During OSD Removal<a class="headerlink" href="#monitoring-osd-state-during-osd-removal" title="Permalink to this heading"></a></h3>
<p>You can query the state of OSD operations during the process of removing OSDs
by running the following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>osd<span class="w"> </span>rm<span class="w"> </span>status</span>
</pre></div></div><p>Expected output:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">OSD_ID</span>  <span class="n">HOST</span>         <span class="n">STATE</span>                    <span class="n">PG_COUNT</span>  <span class="n">REPLACE</span>  <span class="n">FORCE</span>  <span class="n">STARTED_AT</span>
<span class="mi">2</span>       <span class="n">cephadm</span><span class="o">-</span><span class="n">dev</span>  <span class="n">done</span><span class="p">,</span> <span class="n">waiting</span> <span class="k">for</span> <span class="n">purge</span>  <span class="mi">0</span>         <span class="kc">True</span>     <span class="kc">False</span>  <span class="mi">2020</span><span class="o">-</span><span class="mi">07</span><span class="o">-</span><span class="mi">17</span> <span class="mi">13</span><span class="p">:</span><span class="mi">01</span><span class="p">:</span><span class="mf">43.147684</span>
<span class="mi">3</span>       <span class="n">cephadm</span><span class="o">-</span><span class="n">dev</span>  <span class="n">draining</span>                 <span class="mi">17</span>        <span class="kc">False</span>    <span class="kc">True</span>   <span class="mi">2020</span><span class="o">-</span><span class="mi">07</span><span class="o">-</span><span class="mi">17</span> <span class="mi">13</span><span class="p">:</span><span class="mi">01</span><span class="p">:</span><span class="mf">45.162158</span>
<span class="mi">4</span>       <span class="n">cephadm</span><span class="o">-</span><span class="n">dev</span>  <span class="n">started</span>                  <span class="mi">42</span>        <span class="kc">False</span>    <span class="kc">True</span>   <span class="mi">2020</span><span class="o">-</span><span class="mi">07</span><span class="o">-</span><span class="mi">17</span> <span class="mi">13</span><span class="p">:</span><span class="mi">01</span><span class="p">:</span><span class="mf">45.162158</span>
</pre></div>
</div>
<p>When no PGs are left on the OSD, it will be decommissioned and removed from the cluster.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>After removing an OSD, if you wipe the LVM physical volume in the device used by the removed OSD, a new OSD will be created.
For more information on this, read about the <code class="docutils literal notranslate"><span class="pre">unmanaged</span></code> parameter in <a class="reference internal" href="#cephadm-osd-declarative"><span class="std std-ref">Declarative State</span></a>.</p>
</div>
</section>
<section id="stopping-osd-removal">
<h3>Stopping OSD Removal<a class="headerlink" href="#stopping-osd-removal" title="Permalink to this heading"></a></h3>
<p>It is possible to stop queued OSD removals by using the following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>osd<span class="w"> </span>rm<span class="w"> </span>stop<span class="w"> </span>&lt;osd_id<span class="o">(</span>s<span class="o">)</span>&gt;</span>
</pre></div></div><p>Example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>osd<span class="w"> </span>rm<span class="w"> </span>stop<span class="w"> </span><span class="m">4</span></span>
</pre></div></div><p>Expected output:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">Stopped</span> <span class="n">OSD</span><span class="p">(</span><span class="n">s</span><span class="p">)</span> <span class="n">removal</span>
</pre></div>
</div>
<p>This resets the state of the OSD and takes it off the removal queue.</p>
</section>
<section id="replacing-an-osd">
<span id="cephadm-replacing-an-osd"></span><h3>Replacing an OSD<a class="headerlink" href="#replacing-an-osd" title="Permalink to this heading"></a></h3>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>osd<span class="w"> </span>rm<span class="w"> </span>&lt;osd_id<span class="o">(</span>s<span class="o">)</span>&gt;<span class="w"> </span>--replace<span class="w"> </span><span class="o">[</span>--force<span class="o">]</span></span>
</pre></div></div><p>Example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>osd<span class="w"> </span>rm<span class="w"> </span><span class="m">4</span><span class="w"> </span>--replace</span>
</pre></div></div><p>Expected output:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">Scheduled</span> <span class="n">OSD</span><span class="p">(</span><span class="n">s</span><span class="p">)</span> <span class="k">for</span> <span class="n">replacement</span>
</pre></div>
</div>
<p>This follows the same procedure as the procedure in the “Remove OSD” section, with
one exception: the OSD is not permanently removed from the CRUSH hierarchy, but is
instead assigned the <code class="docutils literal notranslate"><span class="pre">destroyed</span></code> flag.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>The new OSD that will replace the removed OSD must be created on the same host
as the OSD that was removed.</p>
</div>
<p><strong>Preserving the OSD ID</strong></p>
<p>The <code class="docutils literal notranslate"><span class="pre">destroyed</span></code> flag is used to determine which OSD IDs will be reused in the
next OSD deployment.</p>
<p>If you use OSDSpecs for OSD deployment, your newly added drives will be assigned
the OSD IDs of their replaced counterparts. This assumes that the new drives
still match the OSDSpecs.</p>
<p>Use the <code class="docutils literal notranslate"><span class="pre">--dry-run</span></code> flag to ensure that the <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span> <span class="pre">apply</span> <span class="pre">osd</span></code>
command will do what you intend. The <code class="docutils literal notranslate"><span class="pre">--dry-run</span></code> flag shows what the
outcome of the command will be without executing any changes. When
you are satisfied that the command will do what you want, run the command
without the <code class="docutils literal notranslate"><span class="pre">--dry-run</span></code> flag.</p>
<div class="admonition tip">
<p class="admonition-title">Tip</p>
<p>The name of your OSDSpec can be retrieved with the command <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span> <span class="pre">ls</span></code></p>
</div>
<p>Alternatively, you can use an OSDSpec file:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>apply<span class="w"> </span>-i<span class="w"> </span>&lt;osd_spec_file&gt;<span class="w"> </span>--dry-run</span>
</pre></div></div><p>Expected output:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">NAME</span>                  <span class="n">HOST</span>  <span class="n">DATA</span>     <span class="n">DB</span> <span class="n">WAL</span>
<span class="o">&lt;</span><span class="n">name_of_osd_spec</span><span class="o">&gt;</span>    <span class="n">node1</span> <span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">vdb</span> <span class="o">-</span>  <span class="o">-</span>
</pre></div>
</div>
<p>When this output reflects your intent, omit the <code class="docutils literal notranslate"><span class="pre">--dry-run</span></code> flag to
execute the deployment.</p>
</section>
<section id="erasing-devices-zapping-devices">
<h3>Erasing Devices (Zapping Devices)<a class="headerlink" href="#erasing-devices-zapping-devices" title="Permalink to this heading"></a></h3>
<p>Erase (zap) a device so that it can be reused. <code class="docutils literal notranslate"><span class="pre">zap</span></code> calls <code class="docutils literal notranslate"><span class="pre">ceph-volume</span>
<span class="pre">zap</span></code> on the remote host.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>device<span class="w"> </span>zap<span class="w"> </span>&lt;hostname&gt;<span class="w"> </span>&lt;path&gt;</span>
</pre></div></div><p>Example command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>orch<span class="w"> </span>device<span class="w"> </span>zap<span class="w"> </span>my_hostname<span class="w"> </span>/dev/sdx</span>
</pre></div></div><div class="admonition note">
<p class="admonition-title">Note</p>
<p>If the <code class="docutils literal notranslate"><span class="pre">unmanaged</span></code> flag is not set, <code class="docutils literal notranslate"><span class="pre">cephadm</span></code> automatically deploys drives that
match the OSDSpec.  For example, if you specify the
<code class="docutils literal notranslate"><span class="pre">all-available-devices</span></code> option when creating OSDs, when you <code class="docutils literal notranslate"><span class="pre">zap</span></code> a
device the <code class="docutils literal notranslate"><span class="pre">cephadm</span></code> orchestrator automatically creates a new OSD on the
device.  To disable this behavior, see <a class="reference internal" href="#cephadm-osd-declarative"><span class="std std-ref">Declarative State</span></a>.</p>
</div>
</section>
</section>
<section id="automatically-tuning-osd-memory">
<span id="osd-autotune"></span><h2>Automatically tuning OSD memory<a class="headerlink" href="#automatically-tuning-osd-memory" title="Permalink to this heading"></a></h2>
<p>OSD daemons will adjust their memory consumption based on the
<a class="reference internal" href="../../../rados/configuration/bluestore-config-ref/#confval-osd_memory_target"><code class="xref std std-confval docutils literal notranslate"><span class="pre">osd_memory_target</span></code></a> config option.  If Ceph is deployed
on dedicated nodes that are not sharing
memory with other services, <code class="docutils literal notranslate"><span class="pre">cephadm</span></code> will automatically adjust the per-OSD
memory consumption target based on the total amount of RAM and the number of deployed
OSDs.  This allows the full use of available memory, and adapts when OSDs or
RAM are added or removed.</p>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>Cephadm sets <code class="docutils literal notranslate"><span class="pre">osd_memory_target_autotune</span></code> to <code class="docutils literal notranslate"><span class="pre">true</span></code> by default which is usually not appropriate for converged architectures, where a given node is used for both Ceph and compute purposes.</p>
</div>
<p><code class="docutils literal notranslate"><span class="pre">Cephadm</span></code> will use a fraction
<code class="xref std std-confval docutils literal notranslate"><span class="pre">mgr/cephadm/autotune_memory_target_ratio</span></code> of available memory,
subtracting memory consumed by non-autotuned daemons (non-OSDs and OSDs for which
<code class="docutils literal notranslate"><span class="pre">osd_memory_target_autotune</span></code> is false), and then divide the balance by the number
of OSDs.</p>
<p>The final targets are reflected in the config database with options like the below:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">WHO</span>   <span class="n">MASK</span>      <span class="n">LEVEL</span>   <span class="n">OPTION</span>              <span class="n">VALUE</span>
<span class="n">osd</span>   <span class="n">host</span><span class="p">:</span><span class="n">foo</span>  <span class="n">basic</span>   <span class="n">osd_memory_target</span>   <span class="mi">126092301926</span>
<span class="n">osd</span>   <span class="n">host</span><span class="p">:</span><span class="n">bar</span>  <span class="n">basic</span>   <span class="n">osd_memory_target</span>   <span class="mi">6442450944</span>
</pre></div>
</div>
<p>Both the limits and the current memory consumed by each daemon are visible from
the <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span> <span class="pre">ps</span></code> output in the <code class="docutils literal notranslate"><span class="pre">MEM</span> <span class="pre">LIMIT</span></code> column:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">NAME</span>        <span class="n">HOST</span>  <span class="n">PORTS</span>  <span class="n">STATUS</span>         <span class="n">REFRESHED</span>  <span class="n">AGE</span>  <span class="n">MEM</span> <span class="n">USED</span>  <span class="n">MEM</span> <span class="n">LIMIT</span>  <span class="n">VERSION</span>                <span class="n">IMAGE</span> <span class="n">ID</span>      <span class="n">CONTAINER</span> <span class="n">ID</span>
<span class="n">osd</span><span class="mf">.1</span>       <span class="n">dael</span>         <span class="n">running</span> <span class="p">(</span><span class="mi">3</span><span class="n">h</span><span class="p">)</span>     <span class="mi">10</span><span class="n">s</span> <span class="n">ago</span>   <span class="mi">3</span><span class="n">h</span>    <span class="mi">72857</span><span class="n">k</span>     <span class="mf">117.4</span><span class="n">G</span>  <span class="mf">17.0.0</span><span class="o">-</span><span class="mi">3781</span><span class="o">-</span><span class="n">gafaed750</span>  <span class="mi">7015</span><span class="n">fda3cd67</span>  <span class="mf">9e183363</span><span class="n">d39c</span>
<span class="n">osd</span><span class="mf">.2</span>       <span class="n">dael</span>         <span class="n">running</span> <span class="p">(</span><span class="mi">81</span><span class="n">m</span><span class="p">)</span>    <span class="mi">10</span><span class="n">s</span> <span class="n">ago</span>  <span class="mi">81</span><span class="n">m</span>    <span class="mi">63989</span><span class="n">k</span>     <span class="mf">117.4</span><span class="n">G</span>  <span class="mf">17.0.0</span><span class="o">-</span><span class="mi">3781</span><span class="o">-</span><span class="n">gafaed750</span>  <span class="mi">7015</span><span class="n">fda3cd67</span>  <span class="mi">1</span><span class="n">f0cc479b051</span>
<span class="n">osd</span><span class="mf">.3</span>       <span class="n">dael</span>         <span class="n">running</span> <span class="p">(</span><span class="mi">62</span><span class="n">m</span><span class="p">)</span>    <span class="mi">10</span><span class="n">s</span> <span class="n">ago</span>  <span class="mi">62</span><span class="n">m</span>    <span class="mi">64071</span><span class="n">k</span>     <span class="mf">117.4</span><span class="n">G</span>  <span class="mf">17.0.0</span><span class="o">-</span><span class="mi">3781</span><span class="o">-</span><span class="n">gafaed750</span>  <span class="mi">7015</span><span class="n">fda3cd67</span>  <span class="n">ac5537492f27</span>
</pre></div>
</div>
<p>To exclude an OSD from memory autotuning, disable the <code class="docutils literal notranslate"><span class="pre">autotune</span></code> option
for that OSD and also set a specific memory target.  For example,</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>config<span class="w"> </span><span class="nb">set</span><span class="w"> </span>osd.123<span class="w"> </span>osd_memory_target_autotune<span class="w"> </span><span class="nb">false</span></span>
<span class="prompt1">ceph<span class="w"> </span>config<span class="w"> </span><span class="nb">set</span><span class="w"> </span>osd.123<span class="w"> </span>osd_memory_target<span class="w"> </span>16G</span>
</pre></div></div></section>
<section id="advanced-osd-service-specifications">
<span id="drivegroups"></span><h2>Advanced OSD Service Specifications<a class="headerlink" href="#advanced-osd-service-specifications" title="Permalink to this heading"></a></h2>
<p><a class="reference internal" href="../#orchestrator-cli-service-spec"><span class="std std-ref">Service Specification</span></a>s of type <code class="docutils literal notranslate"><span class="pre">osd</span></code> provide a way to use the
properties of drives to describe a Ceph cluster’s layout. Service specifications
are an abstraction used to tell Ceph which drives to transform into OSDs
and which configurations to apply to those OSDs.
<a class="reference internal" href="../#orchestrator-cli-service-spec"><span class="std std-ref">Service Specification</span></a>s make it possible to target drives
for transformation into OSDs even when the Ceph cluster operator does not know
the specific device names and paths associated with those disks.</p>
<p><a class="reference internal" href="../#orchestrator-cli-service-spec"><span class="std std-ref">Service Specification</span></a>s make it possible to define a <code class="docutils literal notranslate"><span class="pre">.yaml</span></code>
or <code class="docutils literal notranslate"><span class="pre">.json</span></code> file that can be used to reduce the amount of manual work involved
in creating OSDs.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>We recommend that advanced OSD specs include the <code class="docutils literal notranslate"><span class="pre">service_id</span></code> field.
OSDs created using <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span> <span class="pre">daemon</span> <span class="pre">add</span></code> or <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span> <span class="pre">apply</span> <span class="pre">osd</span>
<span class="pre">--all-available-devices</span></code> are placed in the plain <code class="docutils literal notranslate"><span class="pre">osd</span></code> service. Failing
to include a <code class="docutils literal notranslate"><span class="pre">service_id</span></code> in your OSD spec causes the Ceph cluster to mix
the OSDs from your spec with those OSDs, which can potentially result in the
overwriting of service specs created by <code class="docutils literal notranslate"><span class="pre">cephadm</span></code> to track them. Newer
versions of <code class="docutils literal notranslate"><span class="pre">cephadm</span></code> block OSD specs that
do not include the <code class="docutils literal notranslate"><span class="pre">service_id</span></code>.</p>
</div>
<p>For example, instead of running the following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><style type="text/css">
span.prompt3:before {
  content: "[monitor.1]# ";
}
</style><span class="prompt3">ceph<span class="w"> </span>orch<span class="w"> </span>daemon<span class="w"> </span>add<span class="w"> </span>osd<span class="w"> </span>*&lt;host&gt;*:*&lt;path-to-device&gt;*</span>
</pre></div></div><p>for each device and each host, we can create a <code class="docutils literal notranslate"><span class="pre">.yaml</span></code> or <code class="docutils literal notranslate"><span class="pre">.json</span></code> file that
allows us to describe the layout. Here is the most basic example:</p>
<p>Create a file called (for example) <code class="docutils literal notranslate"><span class="pre">osd_spec.yml</span></code>:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">service_type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">osd</span>
<span class="nt">service_id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">default_drive_group</span><span class="w">  </span><span class="c1"># custom name of the osd spec</span>
<span class="nt">placement</span><span class="p">:</span>
<span class="w">  </span><span class="nt">host_pattern</span><span class="p">:</span><span class="w"> </span><span class="s">&#39;*&#39;</span><span class="w">              </span><span class="c1"># which hosts to target</span>
<span class="nt">spec</span><span class="p">:</span>
<span class="w">  </span><span class="nt">data_devices</span><span class="p">:</span><span class="w">                  </span><span class="c1"># the type of devices you are applying specs to</span>
<span class="w">    </span><span class="nt">all</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span><span class="w">                    </span><span class="c1"># a filter, check below for a full list</span>
</pre></div>
</div>
<p>This means :</p>
<ol class="arabic">
<li><p>Turn any available device (<code class="docutils literal notranslate"><span class="pre">ceph-volume`</span> <span class="pre">decides</span> <span class="pre">which</span> <span class="pre">are</span> <span class="pre">_available_)</span> <span class="pre">into</span> <span class="pre">an</span>
<span class="pre">OSD</span> <span class="pre">on</span> <span class="pre">all</span> <span class="pre">hosts</span> <span class="pre">that</span> <span class="pre">match</span> <span class="pre">the</span> <span class="pre">glob</span> <span class="pre">pattern</span> <span class="pre">'*'.</span> <span class="pre">The</span> <span class="pre">glob</span> <span class="pre">pattern</span> <span class="pre">matches</span>
<span class="pre">registered</span> <span class="pre">hosts</span> <span class="pre">from</span> <span class="pre">`ceph</span> <span class="pre">orch</span> <span class="pre">host</span> <span class="pre">ls`.</span> <span class="pre">See</span>
<span class="pre">:ref:`cephadm-services-placement-by-pattern-matching`</span> <span class="pre">for</span> <span class="pre">more</span> <span class="pre">on</span> <span class="pre">using</span>
<span class="pre">``host_pattern</span></code> matching to use devices for OSDs.</p></li>
<li><p>Pass <code class="docutils literal notranslate"><span class="pre">osd_spec.yml</span></code> to <code class="docutils literal notranslate"><span class="pre">osd</span> <span class="pre">create</span></code> by using the following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt3">ceph<span class="w"> </span>orch<span class="w"> </span>apply<span class="w"> </span>-i<span class="w"> </span>/path/to/osd_spec.yml</span>
</pre></div></div><p>This specification is applied to all the matching hosts to deploy OSDs.</p>
<p>Strategies more complex than the one specified by the <code class="docutils literal notranslate"><span class="pre">all</span></code> filter are
possible. See <a class="reference internal" href="#osd-filters"><span class="std std-ref">Filters</span></a> for details.</p>
<p>A <code class="docutils literal notranslate"><span class="pre">--dry-run</span></code> flag can be passed to the <code class="docutils literal notranslate"><span class="pre">apply</span> <span class="pre">osd</span></code> command to display a
synopsis of the proposed layout.</p>
</li>
</ol>
<p>Example</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt3">ceph<span class="w"> </span>orch<span class="w"> </span>apply<span class="w"> </span>-i<span class="w"> </span>/path/to/osd_spec.yml<span class="w"> </span>--dry-run</span>
</pre></div></div><section id="filters">
<span id="osd-filters"></span><h3>Filters<a class="headerlink" href="#filters" title="Permalink to this heading"></a></h3>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Filters are applied using an <cite>AND</cite> operation by default. This means that a drive
must match all filter criteria to be selected. This behavior can
be adjusted by setting <code class="docutils literal notranslate"><span class="pre">filter_logic:</span> <span class="pre">OR</span></code> in the OSD specification.</p>
</div>
<p>Filters are used to select sets of drives for OSD data or WAL+DB offload based
on various attributes. These attributes are gathered by <code class="docutils literal notranslate"><span class="pre">ceph-volume</span></code>’s drive
inventory. Retrieve these attributes with this command:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>ceph-volume<span class="w"> </span>inventory<span class="w"> </span>&lt;/path/to/drive&gt;
</pre></div>
</div>
<section id="vendor-or-model">
<h4>Vendor or Model<a class="headerlink" href="#vendor-or-model" title="Permalink to this heading"></a></h4>
<p>Specific drives can be targeted by vendor brand, manufacturer) or model (SKU):</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">model</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">drive_model_name</span>
</pre></div>
</div>
<p>or</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">vendor</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">drive_vendor_name</span>
</pre></div>
</div>
</section>
<section id="size">
<h4>Size<a class="headerlink" href="#size" title="Permalink to this heading"></a></h4>
<p>Specific drive capacities can be targeted with <cite>size</cite>:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">size</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">size_spec</span>
</pre></div>
</div>
<section id="size-specs">
<h5>Size specs<a class="headerlink" href="#size-specs" title="Permalink to this heading"></a></h5>
<p>Size specifications can be of the following forms:</p>
<ul class="simple">
<li><p>LOW:HIGH</p></li>
<li><p>:HIGH</p></li>
<li><p>LOW:</p></li>
<li><p>EXACT</p></li>
</ul>
<p>We explore examples below.</p>
<p>To match only drives of an exact capacity:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">size</span><span class="p">:</span><span class="w"> </span><span class="s">&#39;10T&#39;</span>
</pre></div>
</div>
<p>Note that drive capacity is often not an exact multiple of units, so it is
often best practice to match drives within a range of sizes as shown below.
This handles future drives of the same class that may be of a different
model and thus slightly different in size.  Or say you have 10 TB drives
today but may add 16 TB drives next year:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">size</span><span class="p">:</span><span class="w"> </span><span class="s">&#39;10T:40T&#39;</span>
</pre></div>
</div>
<p>To match only drives that are less than or equal to 1701 GB in size:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">size</span><span class="p">:</span><span class="w"> </span><span class="s">&#39;:1701G&#39;</span>
</pre></div>
</div>
<p>To include drives equal to or greater than 666 GB in size:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">size</span><span class="p">:</span><span class="w"> </span><span class="s">&#39;666G:&#39;</span>
</pre></div>
</div>
<p>The supported units of size are Megabyte(M), Gigabyte(G) and Terabyte(T).
The <code class="docutils literal notranslate"><span class="pre">B</span></code> (_byte_) suffix for units is also acceptable: <code class="docutils literal notranslate"><span class="pre">MB</span></code>, <code class="docutils literal notranslate"><span class="pre">GB</span></code>, <code class="docutils literal notranslate"><span class="pre">TB</span></code>.</p>
</section>
</section>
<section id="rotational">
<h4>Rotational<a class="headerlink" href="#rotational" title="Permalink to this heading"></a></h4>
<p>This gates based on the ‘rotational’ attribute of each drive, as indicated by
the kernel.  This attribute is usually as expected for bare HDDs and SSDs
installed in each node.  Exotic or layered device presentations may however
be reported differently than you might expect or desire:</p>
<ul class="simple">
<li><p>Network-accessed SAN LUNs attached to the node</p></li>
<li><p>Composite devices presented by <cite>dCache</cite>, <cite>Bcache</cite>, <cite>OpenCAS</cite>, etc.</p></li>
</ul>
<p>In such cases you may align the kernel’s reporting with your expectations
by adding a <code class="docutils literal notranslate"><span class="pre">udev</span></code> rule to override the default behavior.  The below rule
was used for this purpose to override the <code class="docutils literal notranslate"><span class="pre">rotational</span></code> attribute on OSD
nodes with no local physical drives and only attached SAN LUNs. It is not
intended for deployment in all scenarios; you will have to determine what is
right for your systems.  If by emplacing such a rule you summon eldritch horrors
from beyond spacetime, that’s on you.</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>ACTION==&quot;add|change&quot;, KERNEL==&quot;sd[a-z]*&quot;, ATTR{queue/rotational}=&quot;0&quot;
ACTION==&quot;add|change&quot;, KERNEL==&quot;dm*&quot;, ATTR{queue/rotational}=&quot;0&quot;
ACTION==&quot;add|change&quot;, KERNEL==&quot;nbd*&quot;, ATTR{queue/rotational}=&quot;0&quot;
</pre></div>
</div>
<p>Spec file syntax:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">rotational</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">0 | 1</span>
</pre></div>
</div>
<p><cite>1</cite> to match all drives that the kernel indicates are rotational</p>
<p><cite>0</cite> to match all drives that are non-rotational (SATA, SATA, NVMe SSDs, SAN LUNs, etc)</p>
</section>
<section id="all">
<h4>All<a class="headerlink" href="#all" title="Permalink to this heading"></a></h4>
<p>This matches all drives that are available, i.e. they are free of partitions,
GPT labels, etc.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>This may only be specified for <code class="docutils literal notranslate"><span class="pre">data_devices</span></code>.</p>
</div>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">all</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span>
</pre></div>
</div>
</section>
<section id="limiter">
<h4>Limiter<a class="headerlink" href="#limiter" title="Permalink to this heading"></a></h4>
<p>If filters are specified but you wish to limit the number of drives that they
match, use the <code class="docutils literal notranslate"><span class="pre">limit</span></code> attribute.  This is useful when one uses some
drives for non-Ceph purposes, or when multiple OSD strategies are
intended.</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">limit</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">2</span>
</pre></div>
</div>
<p>For example, when using <code class="docutils literal notranslate"><span class="pre">vendor</span></code> to match all drives branded <code class="docutils literal notranslate"><span class="pre">VendorA</span></code>
but you wish to use at most two of them per host as OSDs, specify a <code class="docutils literal notranslate"><span class="pre">limit</span></code>:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">data_devices</span><span class="p">:</span>
<span class="w">  </span><span class="nt">vendor</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">VendorA</span>
<span class="w">  </span><span class="nt">limit</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">2</span>
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p><code class="docutils literal notranslate"><span class="pre">limit</span></code> is usually appropriate in only certain specific scenarios.</p>
</div>
</section>
</section>
<section id="additional-options">
<h3>Additional Options<a class="headerlink" href="#additional-options" title="Permalink to this heading"></a></h3>
<p>There are multiple optional settings that specify the way OSDs are deployed.
Add these options to an OSD spec for them to take effect.</p>
<p>This example deploys encrypted OSDs on all unused drives.  Note that if Linux
MD mirroring is used for the boot, <cite>/var/log</cite>, or other volumes this spec _may_
grab replacement or added drives before you can employ them for non-OSD purposes.</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">service_type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">osd</span>
<span class="nt">service_id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">example_osd_spec</span>
<span class="nt">placement</span><span class="p">:</span>
<span class="w">  </span><span class="nt">host_pattern</span><span class="p">:</span><span class="w"> </span><span class="s">&#39;*&#39;</span>
<span class="nt">spec</span><span class="p">:</span>
<span class="w">  </span><span class="nt">data_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">all</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span>
<span class="w">  </span><span class="nt">encrypted</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span>
</pre></div>
</div>
<p>Ceph Squid onwards support TPM2 token enrollment for LUKS2 devices.
Add the <cite>tpm2</cite> attribute to the OSD spec:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">service_type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">osd</span>
<span class="nt">service_id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">example_osd_spec_with_tpm2</span>
<span class="nt">placement</span><span class="p">:</span>
<span class="w">  </span><span class="nt">host_pattern</span><span class="p">:</span><span class="w"> </span><span class="s">&#39;*&#39;</span>
<span class="nt">spec</span><span class="p">:</span>
<span class="w">  </span><span class="nt">data_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">all</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span>
<span class="w">  </span><span class="nt">encrypted</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span>
<span class="w">  </span><span class="nt">tpm2</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span>
</pre></div>
</div>
<p>See a full list in the DriveGroupSpecs</p>
<dl class="py class">
<dt class="sig sig-object py" id="ceph.deployment.drive_group.DriveGroupSpec">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">ceph.deployment.drive_group.</span></span><span class="sig-name descname"><span class="pre">DriveGroupSpec</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">placement</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">service_id</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">data_devices</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">db_devices</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">wal_devices</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">journal_devices</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">data_directories</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">osds_per_device</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">objectstore</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'bluestore'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">encrypted</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">tpm2</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">db_slots</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">wal_slots</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">osd_id_claims</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">block_db_size</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">block_wal_size</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">journal_size</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">service_type</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">unmanaged</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">filter_logic</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'AND'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">preview_only</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">extra_container_args</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">extra_entrypoint_args</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">data_allocate_fraction</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">method</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">config</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">custom_configs</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">crush_device_class</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec" title="Permalink to this definition"></a></dt>
<dd><p>Describe a drive group in the same form that ceph-volume
understands.</p>
<dl class="py attribute">
<dt class="sig sig-object py" id="ceph.deployment.drive_group.DriveGroupSpec.block_db_size">
<span class="sig-name descname"><span class="pre">block_db_size</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">int</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">str</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">None</span></em><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.block_db_size" title="Permalink to this definition"></a></dt>
<dd><p>Set (or override) the “bluestore_block_db_size” value, in bytes</p>
</dd></dl>

<dl class="py attribute">
<dt class="sig sig-object py" id="ceph.deployment.drive_group.DriveGroupSpec.block_wal_size">
<span class="sig-name descname"><span class="pre">block_wal_size</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">int</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">str</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">None</span></em><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.block_wal_size" title="Permalink to this definition"></a></dt>
<dd><p>Set (or override) the “bluestore_block_wal_size” value, in bytes</p>
</dd></dl>

<dl class="py attribute">
<dt class="sig sig-object py" id="ceph.deployment.drive_group.DriveGroupSpec.crush_device_class">
<span class="sig-name descname"><span class="pre">crush_device_class</span></span><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.crush_device_class" title="Permalink to this definition"></a></dt>
<dd><p>Crush device class to assign to OSDs</p>
</dd></dl>

<dl class="py attribute">
<dt class="sig sig-object py" id="ceph.deployment.drive_group.DriveGroupSpec.data_allocate_fraction">
<span class="sig-name descname"><span class="pre">data_allocate_fraction</span></span><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.data_allocate_fraction" title="Permalink to this definition"></a></dt>
<dd><p>Allocate a fraction of the data device (0,1.0]</p>
</dd></dl>

<dl class="py attribute">
<dt class="sig sig-object py" id="ceph.deployment.drive_group.DriveGroupSpec.data_devices">
<span class="sig-name descname"><span class="pre">data_devices</span></span><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.data_devices" title="Permalink to this definition"></a></dt>
<dd><p>A <code class="xref py py-class docutils literal notranslate"><span class="pre">ceph.deployment.drive_group.DeviceSelection</span></code></p>
</dd></dl>

<dl class="py attribute">
<dt class="sig sig-object py" id="ceph.deployment.drive_group.DriveGroupSpec.data_directories">
<span class="sig-name descname"><span class="pre">data_directories</span></span><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.data_directories" title="Permalink to this definition"></a></dt>
<dd><p>A list of strings, containing paths which should back OSDs</p>
</dd></dl>

<dl class="py attribute">
<dt class="sig sig-object py" id="ceph.deployment.drive_group.DriveGroupSpec.db_devices">
<span class="sig-name descname"><span class="pre">db_devices</span></span><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.db_devices" title="Permalink to this definition"></a></dt>
<dd><p>A <code class="xref py py-class docutils literal notranslate"><span class="pre">ceph.deployment.drive_group.DeviceSelection</span></code></p>
</dd></dl>

<dl class="py attribute">
<dt class="sig sig-object py" id="ceph.deployment.drive_group.DriveGroupSpec.db_slots">
<span class="sig-name descname"><span class="pre">db_slots</span></span><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.db_slots" title="Permalink to this definition"></a></dt>
<dd><p>How many OSDs per DB device</p>
</dd></dl>

<dl class="py attribute">
<dt class="sig sig-object py" id="ceph.deployment.drive_group.DriveGroupSpec.encrypted">
<span class="sig-name descname"><span class="pre">encrypted</span></span><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.encrypted" title="Permalink to this definition"></a></dt>
<dd><p><code class="docutils literal notranslate"><span class="pre">true</span></code> or <code class="docutils literal notranslate"><span class="pre">false</span></code></p>
</dd></dl>

<dl class="py attribute">
<dt class="sig sig-object py" id="ceph.deployment.drive_group.DriveGroupSpec.filter_logic">
<span class="sig-name descname"><span class="pre">filter_logic</span></span><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.filter_logic" title="Permalink to this definition"></a></dt>
<dd><p>The logic gate we use to match disks with filters.
defaults to ‘AND’</p>
</dd></dl>

<dl class="py attribute">
<dt class="sig sig-object py" id="ceph.deployment.drive_group.DriveGroupSpec.journal_devices">
<span class="sig-name descname"><span class="pre">journal_devices</span></span><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.journal_devices" title="Permalink to this definition"></a></dt>
<dd><p>A <code class="xref py py-class docutils literal notranslate"><span class="pre">ceph.deployment.drive_group.DeviceSelection</span></code></p>
</dd></dl>

<dl class="py attribute">
<dt class="sig sig-object py" id="ceph.deployment.drive_group.DriveGroupSpec.journal_size">
<span class="sig-name descname"><span class="pre">journal_size</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">int</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">str</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">None</span></em><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.journal_size" title="Permalink to this definition"></a></dt>
<dd><p>set journal_size in bytes</p>
</dd></dl>

<dl class="py attribute">
<dt class="sig sig-object py" id="ceph.deployment.drive_group.DriveGroupSpec.objectstore">
<span class="sig-name descname"><span class="pre">objectstore</span></span><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.objectstore" title="Permalink to this definition"></a></dt>
<dd><p><code class="docutils literal notranslate"><span class="pre">filestore</span></code> or <code class="docutils literal notranslate"><span class="pre">bluestore</span></code></p>
</dd></dl>

<dl class="py attribute">
<dt class="sig sig-object py" id="ceph.deployment.drive_group.DriveGroupSpec.osd_id_claims">
<span class="sig-name descname"><span class="pre">osd_id_claims</span></span><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.osd_id_claims" title="Permalink to this definition"></a></dt>
<dd><p>Optional: mapping of host -&gt; List of osd_ids that should be replaced
See <a class="reference internal" href="../../../mgr/orchestrator_modules/#orchestrator-osd-replace"><span class="std std-ref">OSD 替换</span></a></p>
</dd></dl>

<dl class="py attribute">
<dt class="sig sig-object py" id="ceph.deployment.drive_group.DriveGroupSpec.osds_per_device">
<span class="sig-name descname"><span class="pre">osds_per_device</span></span><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.osds_per_device" title="Permalink to this definition"></a></dt>
<dd><p>Number of osd daemons per “DATA” device.
To fully utilize nvme devices multiple osds are required.
Can be used to split dual-actuator devices across 2 OSDs, by setting the option to 2.</p>
</dd></dl>

<dl class="py attribute">
<dt class="sig sig-object py" id="ceph.deployment.drive_group.DriveGroupSpec.preview_only">
<span class="sig-name descname"><span class="pre">preview_only</span></span><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.preview_only" title="Permalink to this definition"></a></dt>
<dd><p>If this should be treated as a ‘preview’ spec</p>
</dd></dl>

<dl class="py attribute">
<dt class="sig sig-object py" id="ceph.deployment.drive_group.DriveGroupSpec.tpm2">
<span class="sig-name descname"><span class="pre">tpm2</span></span><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.tpm2" title="Permalink to this definition"></a></dt>
<dd><p><code class="docutils literal notranslate"><span class="pre">true</span></code> or <code class="docutils literal notranslate"><span class="pre">false</span></code></p>
</dd></dl>

<dl class="py attribute">
<dt class="sig sig-object py" id="ceph.deployment.drive_group.DriveGroupSpec.wal_devices">
<span class="sig-name descname"><span class="pre">wal_devices</span></span><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.wal_devices" title="Permalink to this definition"></a></dt>
<dd><p>A <code class="xref py py-class docutils literal notranslate"><span class="pre">ceph.deployment.drive_group.DeviceSelection</span></code></p>
</dd></dl>

<dl class="py attribute">
<dt class="sig sig-object py" id="ceph.deployment.drive_group.DriveGroupSpec.wal_slots">
<span class="sig-name descname"><span class="pre">wal_slots</span></span><a class="headerlink" href="#ceph.deployment.drive_group.DriveGroupSpec.wal_slots" title="Permalink to this definition"></a></dt>
<dd><p>How many OSDs per WAL device</p>
</dd></dl>

</dd></dl>

</section>
</section>
<section id="examples">
<h2>Examples<a class="headerlink" href="#examples" title="Permalink to this heading"></a></h2>
<section id="the-simple-case">
<h3>The simple case<a class="headerlink" href="#the-simple-case" title="Permalink to this heading"></a></h3>
<p>When all cluster nodes have identical drives and we wish to use
them all as OSDs with offloaded WAL+DB:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>10 HDDs
Vendor: VendorA
Model: HDD-123-foo
Size: 4TB

2 SAS/SATA SSDs
Vendor: VendorB
Model: MC-55-44-ZX
Size: 512GB
</pre></div>
</div>
<p>This is a common arrangement and can be described easily:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">service_type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">osd</span>
<span class="nt">service_id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">osd_spec_default</span>
<span class="nt">placement</span><span class="p">:</span>
<span class="w">  </span><span class="nt">host_pattern</span><span class="p">:</span><span class="w"> </span><span class="s">&#39;*&#39;</span>
<span class="nt">spec</span><span class="p">:</span>
<span class="w">  </span><span class="nt">data_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">model</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">HDD-123-foo</span><span class="w">       </span><span class="c1"># Note, HDD-123 would also be valid</span>
<span class="w">  </span><span class="nt">db_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">model</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">MC-55-44-XZ</span><span class="w">       </span><span class="c1"># Same here, MC-55-44 is valid</span>
</pre></div>
</div>
<p>However, we can improve the OSD specification by filtering based on properties
of the drives instead of specific models, as models may change over time as
drives are replaced or added:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">service_type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">osd</span>
<span class="nt">service_id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">osd_spec_default</span>
<span class="nt">placement</span><span class="p">:</span>
<span class="w">  </span><span class="nt">host_pattern</span><span class="p">:</span><span class="w"> </span><span class="s">&#39;*&#39;</span>
<span class="nt">spec</span><span class="p">:</span>
<span class="w">  </span><span class="nt">data_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">rotational</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">1</span><span class="w">            </span><span class="c1"># The kernel flags as HDD</span>
<span class="w">  </span><span class="nt">db_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">rotational</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">0</span><span class="w">            </span><span class="c1"># The kernel flags as SSD (SAS/SATA/NVMe)</span>
</pre></div>
</div>
<p>Here designate all HDDs to be data devices (OSDs) and all SSDs to be used
for WAL+DB offload.</p>
<p>If you know that drives larger than 2 TB should always be used as data devices,
and drives smaller than 2 TB should always be used as WAL/DB devices, you can
filter by size:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">service_type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">osd</span>
<span class="nt">service_id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">osd_spec_default</span>
<span class="nt">placement</span><span class="p">:</span>
<span class="w">  </span><span class="nt">host_pattern</span><span class="p">:</span><span class="w"> </span><span class="s">&#39;*&#39;</span>
<span class="nt">spec</span><span class="p">:</span>
<span class="w">  </span><span class="nt">data_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">size</span><span class="p">:</span><span class="w"> </span><span class="s">&#39;2TB:&#39;</span><span class="w">           </span><span class="c1"># Drives larger than 2 TB</span>
<span class="w">  </span><span class="nt">db_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">size</span><span class="p">:</span><span class="w"> </span><span class="s">&#39;:2TB&#39;</span><span class="w">           </span><span class="c1"># Drives smaller than 2TB</span>
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>All of the above OSD specs are equally valid. Which you use depends on taste and on how much you expect your node layout to change.</p>
</div>
</section>
<section id="multiple-osd-specs-for-a-single-host">
<h3>Multiple OSD specs for a single host<a class="headerlink" href="#multiple-osd-specs-for-a-single-host" title="Permalink to this heading"></a></h3>
<p>Here we specify two distinct strategies for deploying OSDs across multiple
types of media, usually for use by separate pools:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>10 HDDs
Vendor: VendorA
Model: HDD-123-foo
Size: 4TB

12 SAS/SATA SSDs
Vendor: VendorB
Model: MC-55-44-ZX
Size: 512GB

2 NVME SSDs
Vendor: VendorC
Model: NVME-QQQQ-987
Size: 256GB
</pre></div>
</div>
<ul class="simple">
<li><p>10 HDD OSDs use 2 SATA/SAS SSDs for WAL+DB offload</p></li>
<li><p>10 SATA/SAS SSD OSDs share 2 NVMe SSDs for WAL+DB offload</p></li>
</ul>
<p>This can be specificed with two service specs in the same file:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">service_type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">osd</span>
<span class="nt">service_id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">osd_spec_hdd</span>
<span class="nt">placement</span><span class="p">:</span>
<span class="w">  </span><span class="nt">host_pattern</span><span class="p">:</span><span class="w"> </span><span class="s">&#39;*&#39;</span>
<span class="nt">spec</span><span class="p">:</span>
<span class="w">  </span><span class="nt">data_devices</span><span class="p">:</span><span class="w">             </span><span class="c1"># Select all drives the kernel identifies as HDDs</span>
<span class="w">    </span><span class="nt">rotational</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">1</span><span class="w">           </span><span class="c1">#  for OSD data</span>
<span class="w">  </span><span class="nt">db_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">model</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">MC-55-44-XZ</span><span class="w">      </span><span class="c1"># Select only this model for WAL+DB offload</span>
<span class="w">    </span><span class="nt">limit</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">2</span><span class="w">                </span><span class="c1"># Select at most two for this purpose</span>
<span class="w">    </span><span class="nt">db_slots</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">5</span><span class="w">             </span><span class="c1"># Back five slower HDD data devices with each</span>

<span class="nn">---</span>
<span class="nt">service_type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">osd</span>
<span class="nt">service_id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">osd_spec_ssd</span><span class="w">    </span><span class="c1"># Unique so it doesn&#39;t overwrite the above</span>
<span class="nt">placement</span><span class="p">:</span>
<span class="w">  </span><span class="nt">host_pattern</span><span class="p">:</span><span class="w"> </span><span class="s">&#39;*&#39;</span>
<span class="nt">spec</span><span class="p">:</span>
<span class="w">  </span><span class="nt">data_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">model</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">MC-55-44-XZ</span><span class="w">      </span><span class="c1"># Select drives of this model for OSD data</span>
<span class="w">  </span><span class="nt">db_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">vendor</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">VendorC</span><span class="w">         </span><span class="c1"># Select drives of this brand for WAL+DB</span>
<span class="w">    </span><span class="nt">db_slots</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">2</span><span class="w">             </span><span class="c1"># Back two slower SAS/SATA SSD data devices with each</span>
</pre></div>
</div>
<p>This would create the desired layout by using all HDDs as data devices with two
SATA/SAS SSDs assigned as dedicated DB/WAL devices, each backing five HDD OSDs.
The remaining ten SAS/SATA SSDs will be
used as OSD data devices, with <code class="docutils literal notranslate"><span class="pre">VendorC</span></code> NVMEs SSDs assigned as
dedicated DB/WAL devices, each serving two SAS/SATA OSDs.  We call these _hybrid OSDs.</p>
</section>
<section id="multiple-hosts-with-the-same-disk-layout">
<h3>Multiple hosts with the same disk layout<a class="headerlink" href="#multiple-hosts-with-the-same-disk-layout" title="Permalink to this heading"></a></h3>
<p>When a cluster comprises hosts with different drive layouts, or a complex
constellation of multiple media types, it is recommended to apply
multiple OSD specs, each matching only one set of hosts.
Typically you will have a single spec for each type of host.</p>
<p>The <code class="docutils literal notranslate"><span class="pre">service_id</span></code> must be unique: if a new OSD spec with an already
applied <code class="docutils literal notranslate"><span class="pre">service_id</span></code> is applied, the existing OSD spec will be superseded.
Cephadm will then create new OSD daemons on unused drives based on the new spec
definition. Existing OSD daemons will not be affected. See <a class="reference internal" href="#cephadm-osd-declarative"><span class="std std-ref">Declarative State</span></a>.</p>
<p>Example:</p>
<p>Nodes 1-5:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>20 HDDs
Vendor: VendorA
Model: SSD-123-foo
Size: 4TB
2 SSDs
Vendor: VendorB
Model: MC-55-44-ZX
Size: 512GB
</pre></div>
</div>
<p>Nodes 6-10:</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>5 NVMEs
Vendor: VendorA
Model: SSD-123-foo
Size: 4TB
20 SSDs
Vendor: VendorB
Model: MC-55-44-ZX
Size: 512GB
</pre></div>
</div>
<p>You can specify a <code class="docutils literal notranslate"><span class="pre">placement</span></code> to target only certain nodes.</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">service_type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">osd</span>
<span class="nt">service_id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">disk_layout_a</span>
<span class="nt">placement</span><span class="p">:</span>
<span class="w">  </span><span class="nt">label</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">disk_layout_a</span>
<span class="nt">spec</span><span class="p">:</span>
<span class="w">  </span><span class="nt">data_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">rotational</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">1</span><span class="w">           </span><span class="c1"># All drives identified as HDDs</span>
<span class="w">  </span><span class="nt">db_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">rotational</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">0</span><span class="w">           </span><span class="c1"># All drives identified as SSDs</span>
<span class="nn">---</span>
<span class="nt">service_type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">osd</span>
<span class="nt">service_id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">disk_layout_b</span>
<span class="nt">placement</span><span class="p">:</span>
<span class="w">  </span><span class="nt">label</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">disk_layout_b</span>
<span class="nt">spec</span><span class="p">:</span>
<span class="w">  </span><span class="nt">data_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">model</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">MC-55-44-XZ</span><span class="w">      </span><span class="c1"># Only this model</span>
<span class="w">  </span><span class="nt">db_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">model</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">SSD-123-foo</span><span class="w">      </span><span class="c1"># Only this model</span>
</pre></div>
</div>
<p>This applies different OSD specs to different hosts that match hosts
tagged with <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span></code> labels via the <code class="docutils literal notranslate"><span class="pre">placement</span></code> filter.
See <a class="reference internal" href="../#orchestrator-cli-placement-spec"><span class="std std-ref">Daemon Placement</span></a></p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Assuming each host has a unique disk layout, each OSD
spec must have a unique <code class="docutils literal notranslate"><span class="pre">service_id</span></code>.</p>
</div>
</section>
<section id="dedicated-wal-db">
<h3>Dedicated WAL + DB<a class="headerlink" href="#dedicated-wal-db" title="Permalink to this heading"></a></h3>
<p>All previous cases colocated the WALs with the DBs.
It is however possible to deploy the WAL on a separate device if desired.</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>20 HDDs
Vendor: VendorA
Model: SSD-123-foo
Size: 4TB

2 SAS/SATA SSDs
Vendor: VendorB
Model: MC-55-44-ZX
Size: 512GB

2 NVME SSDs
Vendor: VendorC
Model: NVME-QQQQ-987
Size: 256GB
</pre></div>
</div>
<p>The OSD spec for this case would look like the following, using the <code class="docutils literal notranslate"><span class="pre">model</span></code> filter:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">service_type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">osd</span>
<span class="nt">service_id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">osd_spec_default</span>
<span class="nt">placement</span><span class="p">:</span>
<span class="w">  </span><span class="nt">host_pattern</span><span class="p">:</span><span class="w"> </span><span class="s">&#39;*&#39;</span>
<span class="nt">spec</span><span class="p">:</span>
<span class="w">  </span><span class="nt">data_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">model</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">MC-55-44-XZ</span>
<span class="w">  </span><span class="nt">db_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">model</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">SSD-123-foo</span>
<span class="w">  </span><span class="nt">wal_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">model</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">NVME-QQQQ-987</span>
</pre></div>
</div>
<p>It is also possible to specify device paths as below, when every matched host
is expected to present devices identically.</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">service_type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">osd</span>
<span class="nt">service_id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">osd_using_paths</span>
<span class="nt">placement</span><span class="p">:</span>
<span class="w">  </span><span class="nt">hosts</span><span class="p">:</span>
<span class="w">    </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">node01</span>
<span class="w">    </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">node02</span>
<span class="nt">spec</span><span class="p">:</span>
<span class="w">  </span><span class="nt">data_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">paths</span><span class="p">:</span>
<span class="w">    </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/dev/sdb</span>
<span class="w">  </span><span class="nt">db_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">paths</span><span class="p">:</span>
<span class="w">    </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/dev/sdc</span>
<span class="w">  </span><span class="nt">wal_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">paths</span><span class="p">:</span>
<span class="w">    </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/dev/sdd</span>
</pre></div>
</div>
<p>In most cases it is preferable to accomplish this with other filters
including <code class="docutils literal notranslate"><span class="pre">size</span></code> or <code class="docutils literal notranslate"><span class="pre">vendor</span></code> so that OSD services adapt when
Linux or an HBA may enumerate devices differently across boots, or when
drives are added or replaced.</p>
<p>It is possible to specify a <code class="docutils literal notranslate"><span class="pre">crush_device_class</span></code> parameter
to be applied to OSDs created on devices matched by the <code class="docutils literal notranslate"><span class="pre">paths</span></code> filter:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">service_type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">osd</span>
<span class="nt">service_id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">osd_using_paths</span>
<span class="nt">placement</span><span class="p">:</span>
<span class="w">  </span><span class="nt">hosts</span><span class="p">:</span>
<span class="w">    </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">node01</span>
<span class="w">    </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">node02</span>
<span class="nt">crush_device_class</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ssd</span>
<span class="nt">spec</span><span class="p">:</span>
<span class="w">  </span><span class="nt">data_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">paths</span><span class="p">:</span>
<span class="w">    </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/dev/sdb</span>
<span class="w">    </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/dev/sdc</span>
<span class="w">  </span><span class="nt">db_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">paths</span><span class="p">:</span>
<span class="w">    </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/dev/sdd</span>
<span class="w">  </span><span class="nt">wal_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">paths</span><span class="p">:</span>
<span class="w">    </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/dev/sde</span>
</pre></div>
</div>
<p>The <code class="docutils literal notranslate"><span class="pre">crush_device_class</span></code> attribute may be specified at OSD granularity
via the <code class="docutils literal notranslate"><span class="pre">paths</span></code> keyword with the following syntax:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">service_type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">osd</span>
<span class="nt">service_id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">osd_using_paths</span>
<span class="nt">placement</span><span class="p">:</span>
<span class="w">  </span><span class="nt">hosts</span><span class="p">:</span>
<span class="w">    </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">node01</span>
<span class="w">    </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">node02</span>
<span class="nt">crush_device_class</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ssd</span>
<span class="nt">spec</span><span class="p">:</span>
<span class="w">  </span><span class="nt">data_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">paths</span><span class="p">:</span>
<span class="w">    </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/dev/sdb</span>
<span class="w">      </span><span class="nt">crush_device_class</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ssd</span>
<span class="w">    </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/dev/sdc</span>
<span class="w">      </span><span class="nt">crush_device_class</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">nvme</span>
<span class="w">  </span><span class="nt">db_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">paths</span><span class="p">:</span>
<span class="w">    </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/dev/sdd</span>
<span class="w">  </span><span class="nt">wal_devices</span><span class="p">:</span>
<span class="w">    </span><span class="nt">paths</span><span class="p">:</span>
<span class="w">    </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/dev/sde</span>
</pre></div>
</div>
</section>
</section>
<section id="activate-existing-osds">
<span id="cephadm-osd-activate"></span><h2>Activate existing OSDs<a class="headerlink" href="#activate-existing-osds" title="Permalink to this heading"></a></h2>
<p>If a host’s operating system has been reinstalled, existing OSDs
must be activated again. <code class="docutils literal notranslate"><span class="pre">cephadm</span></code> provides a wrapper for
<a class="reference internal" href="../../../ceph-volume/lvm/activate/#ceph-volume-lvm-activate"><span class="std std-ref">activate</span></a> that activates all existing OSDs on a host.</p>
<p>The following procedure explains how to use <code class="docutils literal notranslate"><span class="pre">cephadm</span></code> to activate OSDs on a
host that has had its operating system reinstalled.</p>
<p>This example applies to two hosts: <code class="docutils literal notranslate"><span class="pre">ceph01</span></code> and <code class="docutils literal notranslate"><span class="pre">ceph04</span></code>.</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">ceph01</span></code> is a host equipped with an admin keyring.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">ceph04</span></code> is the host with the recently reinstalled operating system.</p></li>
</ul>
<ol class="arabic">
<li><p>Install <code class="docutils literal notranslate"><span class="pre">cephadm</span></code> and <code class="docutils literal notranslate"><span class="pre">podman</span></code> on the host. The command for installing
these utilities will depend upon the operating system of the host.</p></li>
<li><p>Retrieve the public key.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><style type="text/css">
span.prompt4:before {
  content: "ceph01# ";
}
</style><span class="prompt4"><span class="nb">cd</span><span class="w"> </span>/tmp<span class="w"> </span><span class="p">;</span><span class="w"> </span>ceph<span class="w"> </span>cephadm<span class="w"> </span>get-pub-key<span class="w"> </span>&gt;<span class="w"> </span>ceph.pub</span>
</pre></div></div></li>
<li><p>Copy the key (from <code class="docutils literal notranslate"><span class="pre">ceph01</span></code>) to the freshly reinstalled host (<code class="docutils literal notranslate"><span class="pre">ceph04</span></code>):</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt4">ssh-copy-id<span class="w"> </span>-f<span class="w"> </span>-i<span class="w"> </span>ceph.pub<span class="w"> </span>root@&lt;hostname&gt;</span>
</pre></div></div></li>
<li><p>Retrieve the private key in order to test the connection:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt4"><span class="nb">cd</span><span class="w"> </span>/tmp<span class="w"> </span><span class="p">;</span><span class="w"> </span>ceph<span class="w"> </span>config-key<span class="w"> </span>get<span class="w"> </span>mgr/cephadm/ssh_identity_key<span class="w"> </span>&gt;<span class="w"> </span>ceph-private.key</span>
</pre></div></div></li>
<li><p>From <code class="docutils literal notranslate"><span class="pre">ceph01</span></code>, modify the permissions of <code class="docutils literal notranslate"><span class="pre">ceph-private.key</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt4">chmod<span class="w"> </span><span class="m">400</span><span class="w"> </span>ceph-private.key</span>
</pre></div></div></li>
<li><p>Log in to <code class="docutils literal notranslate"><span class="pre">ceph04</span></code> from <code class="docutils literal notranslate"><span class="pre">ceph01</span></code> to test the connection and
configuration:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt4">ssh<span class="w"> </span>-i<span class="w"> </span>/tmp/ceph-private.key<span class="w"> </span>ceph04</span>
</pre></div></div></li>
<li><p>While logged into <code class="docutils literal notranslate"><span class="pre">ceph01</span></code>, remove <code class="docutils literal notranslate"><span class="pre">ceph.pub</span></code> and <code class="docutils literal notranslate"><span class="pre">ceph-private.key</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt4"><span class="nb">cd</span><span class="w"> </span>/tmp<span class="w"> </span><span class="p">;</span><span class="w"> </span>rm<span class="w"> </span>ceph.pub<span class="w"> </span>ceph-private.key</span>
</pre></div></div></li>
<li><p>If you run your own container registry, instruct the orchestrator to log into
it on each host:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>cephadm<span class="w"> </span>registry-login<span class="w"> </span>my-registry.domain<span class="w"> </span>&lt;user&gt;<span class="w"> </span>&lt;password&gt;</span>
</pre></div></div><p>When the orchestrator performs the registry login, it will attempt to deploy
any missing daemons to the host. This includes <code class="docutils literal notranslate"><span class="pre">crash</span></code>, <code class="docutils literal notranslate"><span class="pre">node-exporter</span></code>,
and any other daemons that the host ran before its operating system was
reinstalled.</p>
<p>To be clea: <code class="docutils literal notranslate"><span class="pre">cephadm</span></code> attempts to deploy missing daemons to all
hosts managed by cephadm, when <code class="docutils literal notranslate"><span class="pre">cephadm</span></code>
determines that the hosts are online. In this context, “online” means
“present in the output of the <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span> <span class="pre">host</span> <span class="pre">ls</span></code> command and with a
status that is not <code class="docutils literal notranslate"><span class="pre">offline</span></code> or <code class="docutils literal notranslate"><span class="pre">maintenance</span></code>. If it is necessary to log
in to the registry in order to pull the images for the missing daemons, then
deployment of the missing daemons will fail until the process of logging
in to the registry has been completed.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>This step is not necessary if you do not run your own container
registry. If your host is still in the “host list”, which can be
retrieved by running the command <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">orch</span> <span class="pre">host</span> <span class="pre">ls</span></code>, you do not
need to run this command.</p>
</div>
</li>
<li><p>Activate the OSDs on the host that has recently had its operating system
reinstalled:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>cephadm<span class="w"> </span>osd<span class="w"> </span>activate<span class="w"> </span>ceph04</span>
</pre></div></div><p>This command causes <code class="docutils literal notranslate"><span class="pre">cephadm</span></code> to scan all existing disks for OSDs. This
command will make <code class="docutils literal notranslate"><span class="pre">cephadm</span></code> deploy any missing daemons to the host
specified.</p>
</li>
</ol>
<p><em>This procedure was developed by Eugen Block in Feburary of 2025, and a blog
post pertinent to its development can be seen here:</em>
<a class="reference external" href="https://heiterbiswolkig.blogs.nde.ag/2025/02/06/cephadm-activate-existing-osds/">Eugen Block’s “Cephadm: Activate existing OSDs” blog post</a>.</p>
</section>
<section id="further-reading">
<h2>Further Reading<a class="headerlink" href="#further-reading" title="Permalink to this heading"></a></h2>
<ul class="simple">
<li><p><a class="reference internal" href="../../../ceph-volume/#ceph-volume"><span class="std std-ref">ceph-volume</span></a></p></li>
<li><p><a class="reference internal" href="../../../rados/#rados-index"><span class="std std-ref">Ceph 存储集群</span></a></p></li>
</ul>
</section>
</section>



<div id="support-the-ceph-foundation" class="admonition note">
  <p class="first admonition-title">Brought to you by the Ceph Foundation</p>
  <p class="last">The Ceph Documentation is a community resource funded and hosted by the non-profit <a href="https://ceph.io/en/foundation/">Ceph Foundation</a>. If you would like to support this and our other efforts, please consider <a href="https://ceph.io/en/foundation/join/">joining now</a>.</p>
</div>


           </div>
           
          </div>
          <footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
        <a href="../mgr/" class="btn btn-neutral float-left" title="MGR Service" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
        <a href="../rgw/" class="btn btn-neutral float-right" title="RGW Service" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>&#169; Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).</p>
  </div>

   

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>